Method and apparatus for motion estimation and motion compensation in video image data

ABSTRACT

The present invention is related to a method and an apparatus for motion estimation and motion compensation in video image data and especially for line based field rate up-conversion motion estimation and motion compensation of video image data.

FIELD OF THE INVENTION

The present invention is related to a method and an apparatus for motionestimation and motion compensation in video image data and especiallyfor line based field rate up-conversion motion estimation and motioncompensation of video image data. The present invention is furtherrelated to a computer program product and a data carrier comprising acomputer program.

TECHNICAL BACKGROUND OF THE INVENTION

The present invention relates to a motion estimation and compensationdevice and more particularly to a motion estimation and motioncompensation device that estimates motion vectors and performs motioncompensated predictions of an interlaced and non interlaced sequence ofchrominance sub-sampled and non sub-sampled video frames.

Hereinafter, the present invention and its underlying problem isdescribed with regard to the processing of a video signal for line-basedmotion estimation and motion compensation within a video processingapparatus such as a microprocessor or microcontroller having line memorydevices, whereas, it should be noted, that the present invention is notrestricted to this application, but can also be used for other videoprocessing apparatus.

The market introduction of TV-sets based on 100/120 Hz frame rate oreven higher required the development of reliable Field/Frame RateUp-conversion (FRU) techniques to remove artefacts within a picture suchas large area flickers and line flickers. Standard FRU methods, whichinterpolate the missing image fields to be displayed on Displays withoutperforming an estimation and compensation of the motion of movingobjects in successive image fields are satisfactory in manyapplications, especially with regard to a better quality of the imageand with regard to the reduction of the above-mentioned artefacts.However, many pictures contain moving objects, like persons, subtitlesand the like, which cause so-called motion judders.

This problem is better understood by referring to FIG. 1, wherein themotion trajectory of the moving objects (white squares) in the originalimage fields (i.e. transmitted and received image fields) is supposed tobe straight-lined. If the missing fields/frames result frominterpolation by means of the above mentioned standard FRU methods (i.e.without motion estimation and compensation), the motion of the movingobject in the interpolated fields (dark grey squares) is not at aposition as expected by the observer (dotted squares). Such artefactsare visible and induce a blurring effect especially of fast movingobjects. These blurring effects typically reduce the quality of thedisplayed images significantly.

In order to avoid such blurring effects and to reduce artefacts severalmethods for motion estimation and motion compensation—or shortlyMEMC—are proposed. This MEMC provides the detecting of a moving part orobject within the received image fields and then the interpolation ofthe missing fields according to the estimated motion by incorporatingthe missing object or part in an estimated field.

FIG. 2 shows schematically the change of the position of a moving objectbetween two successive image fields. Between two successive receivedimage fields/frames, the moving objects will have changed theirposition, e. g. object MO which is in the previous field/frame T inposition A is then in the current field/frame T+1 then in position B.This means, that a motion exists from the previous field/frame T to thecurrent field/frame T+1.

This motion of an object in successive image fields/frames can berepresented by a so-called motion vector. The motion vector ABrepresents the motion of the object MO from position A in the previousfield T to position B in the current field/frame T+1. This motion vectorAB typically has a horizontal and a vertical vector component. Startingfrom point A in the previous field T and applying this motion vector ABto the object MO the object MO is then translated in position B in thecurrent field/frame T+1. The missing position I of the object MO in themissing field/frame T+½ that has to be interpolated must be calculatedby the interpolation of the previous field T and the current field T+1taken account of the respective positions A, B of the moving object MO.If the object MO does not change its position between the previousfield/frame and the current field/frame, e. g. if A and B are the same,position I in the missing field is obtained by the translation of A witha motion vector |AB|/2. In this manner the missing field T+½ isinterpolated with a moving object in the right position with theconsequence that blurring effects are effectively avoided.

Theoretically, for each pixel of a field a corresponding motion vectorhas to be calculated. However, this would increase the number ofcalculation needed and thus the memory requirements enormously. Toreduce this enormous calculation and memory effort there exist basicallytwo different approaches:

The first approach employs a so-called block-based MEMC. This firstapproach assumes that the dimension of the object in the image is alwayslarger than that of a single pixel. Therefore, the image field isdivided into several image blocks. For MEMC only one motion vector iscalculated for each block.

The second approach employs a so-called line-based MEMC. In this secondapproach the algorithm is based on a reduced set of video input data ofa single line of a field or a part of this line. The present inventionis based on this second MEMC approach.

In present line-based MEMC systems, image data is usually stored in alocal buffer or on chip memory, the so-called line memory, to whichrather extreme bandwidth requirements are made. Many present MEMCsystems, like the implementations described by Gerard de Haan in EP 765571 B1 and U.S. Pat. No. 6,034,734, apply a cashe memory (e.g. atwo-dimensional buffer) to reduce the bandwidth requirements and tostore a sub-set of an image. The motion compensation device fetchesvideo image data from this cashe while applying motion vectors.Typically, in MEMC systems this cashe covers the whole search range ofthe motion vectors. Usually, the cashe consists of a great amount ofso-called line memories. This results in a relatively large amount ofmemory, e.g. 720 pixels wide and 24 lines (with an associated maximumvertical vector range of [−12-+12]. Such a cashe comprising a greatamount of single line memories requires a huge memory needed only forMEMC data buffering. As a consequence, the memory portion within theprocessor covers a relatively sizable chip area.

Commonly used MEMC algorithms compensate the motion in two directions,i.e. the motion in the horizontal direction and as well in the verticaldirection. For that operation a memory access should be randomlypossible, which requires for an application in hardware sufficientembedded chip memory within the video processor for the differenttemporal incoming data streams. The size of this embedded chip memorystrongly depends on the search range (i.e. search area) for the motionof an object, as already outlined above, where the motion estimation canmatch similar video patterns in two temporal positions and derive thevelocity of the motion in terms of pixels per frame or per field.

However, this matching process does not always work perfectly, sincemethods to determine the quality of the measured motion vector arerequired. Therefore, for the internal storage of further temporalincoming video signals additional memory resources are required. This,however, increases the amount of embedded memory even further, whichleads to an increase of the chip area since for an integrated circuit itis the chip internal memory which significantly determines the chiparea. Consequently, the chip is getting more and more expensive.Especially in the mainstream market segment such as for modern Plasma-and LCD-TVs these additional costs typically form a limiting factor foran MEMC implementation.

Therefore, there is a need for an efficient internal memory included ina motion estimation device to store data from a reference frame.

The present invention is, therefore, based on the object to provide amore efficient use of the chip-internal resources and especially of thechip-internal memory with regard to motion estimation and motioncompensation.

SUMMARY OF THE INVENTION

In accordance with the present invention, a method comprising thefeatures of claim 1 and/or an apparatus comprising the features of claim16 and/or a computer program product comprising the features of claim 22and/or a data carrier comprising the features of claim 23 is/areprovided.

Accordingly, it is provided:

A method for line-based motion estimation and motion compensation invideo image data, especially for motion estimated and compensated fieldrate up-conversion in consecutive frames of a motion picture, comprisingthe steps of: providing a video signal comprising video image data of aline or part of the line of the picture; buffering the video image datain at least one line memory having the size of one video line or atleast of the incoming or actually processing video image data.

An apparatus for line-based motion estimation and motion compensation invideo image data, especially for motion estimated and compensated fieldrate up-conversion in consecutive frames or fields of a motion picture,wherein the apparatus is configured to perform a method according to thepresent invention.

A computer program product comprising a code, said code being configuredto implement a method according to the present invention.

A data carrier comprising a computer program product ac-cording to thepresent invention.

One basic idea of the present invention is based on the conclusion thatfor the main stream market the performance and therefore the searchrange can be limited to the occurrence of the most likely motiondirection in natural captured scenes, since most of the motion innatural scenes has only one direction. The present invention describes amethod for motion estimation and motion compensation which operates onlyin one direction and therefore performs the motion estimation and motioncompensation operations using a single line buffer memory, the so-calledline memory. This offers the possibility to reduce the chip embeddedmemory to one single line memory for the previous and one single linememory for the current field or frame. This advantageously enablessignificant silicon area reducing and cost saving implementations.

In a preferred embodiment of the present invention, the MEMC is limitedto motion in the horizontal direction only, since most of the motion innatural scenes has this direction.

In video signal processing line memories are often used in otherapplications which already have access to the previous and currentmotion portrayal, e.g. like so-called de-interlacer applications ortemporal noise reduction applications. In a preferred embodiment thesealready existing line memories of the video application are nowadditionally used also for MEMC operations. By using existing linememories of the video signal processing system, no further memorybandwidth has to be added to the memory bus and the memory bandwidth iskept uninfluenced.

Thus, this solution offers the possibility to accomplish the MEMCoperations by adding a minimal or in the optimal case no additionalmemory to the video processing system.

The present invention allows a variable performance of matching ofaccumulation profiles, where beside a comparison of the whole profilealso (shift like) matching of subparts of the accumulation profiles isenabled. The later way represents a kind of fine grinned matchingincreasing accuracy of the matching process and thus of the derivedvertical motion vector.

Additionally, a set of vector candidates derived from the previousaccumulation profile can be used in the matching. Thus, the efficiencyof performance can be increased, as only a set of vector candidates hasto be checked.

A further advantage is that several error reduction methods can beemployed to reduce the visibility of artefacts.

Advantages, embodiments and further developments of the presentinvention can be found in the further subclaims and in the followingdescription, referring to the drawings.

In a preferred embodiment the method comprises the steps of: performingthe motion estimation by detecting and analysing the video image dataand by deriving a motion vector depending on the detected motion,wherein the motion vector contains only motion data of a motion of anobject in one direction and especially in the horizontal direction;using the motion vector for motion compensation to interpolate apicture.

In a preferred embodiment the image data of the previous frame isderived from a first line memory and image data of the current frame isderived from a second line memory.

In a preferred embodiment the first line memory and/or the second linememory are further used in a de-interlacer application and/or a temporalnoise reduction application.

In a preferred embodiment for the motion estimation a matching processis employed, wherein the luminance profile and/or chrominance profileis/are used as matching parameter.

In a preferred embodiment the matching process comprises the steps of:providing a first series of pixels within a line of a previous frame orfield; providing several adjacent second series of pixels within a lineof a current frame or field around a given centre pixel for which themotion shall be determined; successively comparing luminance and/orchrominance values of the second series of pixels with the first seriesof pixels; assigning a quality degree and/or a failure degree to eachsecond series of pixels; selecting that second series of pixels havingthe highest quality-degrees and/or the lowest failure degrees to be themost probable second series of pixels; computing the motion vector inthe horizontal direction on the basis of the selected second series ofpixels.

In a preferred embodiment a SAD-based method and/or an ADRC-based methodare used for the comparison of the luminance and/or chrominance values.

In a preferred embodiment pre-selected samples of motion vectors areused for the motion estimation.

In a preferred embodiment a variation of at least one selected sample ofthe motion vectors is performed in order to set up the process of motionestimation and/or to follow the deviation from the constant motion.

In a preferred embodiment the selection of the selected motion vectorsis treated differently for the first line of a frame or a field comparedto the other lines of the same frame or field.

In a preferred embodiment a damping value which depends on the selectedmotion vector is used to control those motion vectors with similarfailure values and/or to bring the selection process of the motionvector into a desired direction.

In a preferred embodiment a motion vector histogram is used to provide amotion vector ranking on the basis of most and less used motion vectorsin a current frame or field.

In a preferred embodiment for detecting unreliable motion vectors themethod comprises the steps of: Comparing a luminance pixel value of amotion vector compensated previous frame or field with the luminancepixel value of a motion vector compensated current frame or field;Selecting the current motion vector as an unreliable vector if theevaluated difference in the comparing step exceeds a given thresholdvalue; Marking the selected unreliable vector and/or replacing theunreliable vector by a reliable vector.

In a preferred embodiment artefacts generated by remaining and notreplaced unreliable vectors are covered by an intentional blurring ofthose regions which comprises this artefact.

In a preferred embodiment an additional vertical motion estimation andmotion compensation process is performed independently of the horizontalmotion estimation and motion compensation process and/or in combinationwith the horizontal motion estimation and motion compensation process.

In a preferred embodiment the apparatus comprises a first line memoryfor storing image data of the previous frame and a second line memoryfor storing image data of the current frame.

In a preferred embodiment the first line memory and/or the current linememory are configured to be further used in a de-interlacer deviceand/or a temporal noise reduction device.

In a preferred embodiment the apparatus further comprises: an inputterminal for providing a video signal comprising video image data; abuffer for buffering the video image data, wherein the buffer comprisesat least one line memory having at least the size of one video line orat least of the incoming or actually processing video image data; amotion estimation and compensation device for performing the line-basedmotion estimation and motion compensation; an output terminal forproviding a motion compensated video output signal.

In a preferred embodiment a motion vector memory, especially a thirdline memory, is provided to store at least one motion vector. However,the motion vectors may also be stored in external memory devices.

In a preferred embodiment the motion estimation and compensation devicefurther comprises: A motion estimation device which generates a motionvector out of the image data stored in the first and second linememories; A motion compensation device which performs a motioncompensation using the image data stored in the first and second linememories and employing the vector data.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present invention andadvantages thereof, reference is now made to the following descriptiontaken in conjunction with the accompanying drawings. The invention isexplained in more detail below using exemplary embodiments which areschematically specified in the figures of the drawings, in which:

FIG. 1 shows the result of a standard (i.e. non motion compensated) FRUmethod;

FIG. 2 shows the change of position of a moving object between twoconsecutive received image fields;

FIGS. 3A, 3B show the motion estimation principle for the line-basedmotion estimation by means of a current frame and the correspondingprevious frame;

FIG. 4 shows a block diagram of a first embodiment of a line-based MEMCsystem according to the present invention;

FIG. 5 shows an example to illustrate the matching process of the motionestimation;

FIG. 6 shows the basic principle for the provision of a motion vectorhistogram;

FIG. 7 shows an example of a flow diagram to illustrate the process forthe detection of unreliable vectors;

FIG. 8 shows an example of a flow diagram to illustrate the searchprocess for bad vector replacement and error concealment;

FIG. 9 shows by means of three illustrations (A)-(C) examples toillustrate the artefact concealment according to the present invention;

FIG. 10 shows a block diagram illustrating an embodiment of theline-based motion estimation according to the present invention;

FIG. 11 shows a block diagram illustrating an embodiment of theline-based motion compensation according to the present invention usingadaptive artefact concealments;

FIG. 12 shows a block diagram of a second embodiment of a line-basedMEMC system according to the present invention using the line memoriesassigned to the de-interlacer device also for the motion estimationdevice;

FIG. 13 shows a block diagram of a third embodiment of a line-based MEMCsystem according to the present invention which combines the line-basedMEMC with non-vector based-up conversion methods.

In all figures of the drawings elements, features and signals which arethe same or at least have the same functionality have been provided withthe same reference symbols, descriptions and abbreviations unlessexplicitly stated otherwise.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE PRESENT INVENTION

In the following description of the present invention first of all ashort overview of the motion estimation and motion compensation ispresented.

The MEMC method consists mainly of two sections, the motion estimationand the motion compensation method. The motion estimation performs themeasurement of the motion and derives the velocity of the displayedregions in pixel per picture (i.e. field or frame). Also the directionof the motion will be indicated by a positive or negative sign. Thesemeasured motion information is described in the form of a motion vector.The motion vector is used for the motion compensation to interpolate thepicture at the temporal accurate position and to avoid so-called juddereffects and/or so-called motion blurring effects.

FIGS. 3A, 3B show the motion estimation principle for the line-basedmotion estimation by means of a current picture (field or frame) 10 (n)and the corresponding previous picture 11 (n−1). According to thetemporal positions the motion vector 12, 13 will be split by its lengthinto two parts, where the first vector part 12 points into the previouspicture 11 and the second vector part 13 points into the current picture10. For the interpolation of a missing picture 14 (n−½) between thecurrent and the previous pictures 10, 11 pixels 15 from both temporalpictures 10, 11 are taken into account for the compensation. Inline-based MEMC only the pixels 15 within the same line 16 are used atthe same time and the MEMC is performed for a single line 16 of a fieldor frame only. For this kind of MEMC the pixels 15 of the currentpicture 10 are compared with the corresponding pixels 15 of the previouspicture 11 to estimate and compensate the corresponding pixels 15 of themissing picture 14.

FIG. 4 shows a block diagram of a line-based MEMC system according tothe present invention. The MEMC system is denoted by reference number20. The MEMC system 20 comprises an input terminal 21, a bus 22, twoline memories 23, 24, a motion estimation device 25, a motioncompensation device 26 and an output terminal 27. It is assumed that thebus 22 is an external bus 22 and especially an external memory bus 22.However, it may also be possible, that the bus 22 is an internal bus 22.At the input side the bus 22 is connected to an external memory 28device such as a SDRAM, a DDR-RAM, etc. Image data to be displayed in apanel 29 such as a plasma- or LCD-panel or a CRT-screen is stored inthis external memory 28. Via the input terminal 21 and the memory bus 22this image data X1, X1′ is transferred to both line memories 23, 24.According to one embodiment of the present invention only two linememories 23, 24 are needed whereas the first line memory 23 is used forbuffering image data X1 of the previous picture and the other linememory 24 is used for storing the image data X1′ of the current picture.However, it should be clear that also a greater number of line memoriesmay be provided.

A line memory 23, 24 as used in the present patent application indicatesan embedded memory of a size of one video line of a frame or a field orat least less of the incoming video signal stream or actually processingvideo signal stream. A field denotes a video image or picture whichcomprises either odd or even lines. A frame denotes a video imagecomprising of the complete video information for one picture, i.e. of afield for the odd lines and the corresponding field for the even lines.A line denotes a full horizontal row within a field of one video pictureor at least a part of this row.

Both of the line memories 23, 24 are coupled—on their output sides—tothe motion estimation device 25 and to the motion compensation device26. This enables the image data X1, X1′ stored in the line memories 23,24 to be transferred to the motion estimation device 25 and to themotion compensation device 26, respectively. In FIG. 4 the correspondingdata signals to the motion estimation device 25 are denoted by X2, X2′and the corresponding data signals motion compensation device 26 aredenoted by X3, X3′.

The motion estimation device 25 generates a motion vector signal X4 outof the image data X2, X2′ stored in the line memories 23, 24 byemploying a matching process. This vector signal X4 is transferred tothe motion compensation device 26. The motion compensation device 26performs a motion compensation using the image data X3, X3′ stored inthe line memories 23, 24 and applying the vector data X4 to this imagedata X3, X3′. At the output terminal 27, the motion compensation device27 provides a video signal X5 which comprises information for a motioncompensated picture. This video signal X5 is transferred via the outputterminal 27 to a display 29, such as a LCD-panel 29 or the like.

With regard to FIG. 5, hereinafter the operation of the motionestimation device 25 is described in more detail:

For the motion estimation a matching process is employed to select acorresponding series of pixels 32 which fits best to a given amount ofpixels 30. For this selection a given amount of pixels 30 of a line of acurrent frame around the centre pixel 31 for which the motion shall bedetermined is taken from a line memory 24 of the current frame 32.Hereinafter this given amount of pixels 30 is denoted to as series ofpixels 30. In the present embodiment a series of pixels 30 comprises 9single pixels 33. It is self-understood that a series can also comprisea greater or a smaller amount of pixels 33.

For the selection the luminance profile of the pixels 33 is used as thematching parameter. Luminance is a photometric measure of the density ofluminous intensity in a given direction. It describes the amount oflight that passes through or is emitted from a particular area, andfalls within a given solid angle. Thus, luminance is the photometricmeasure of the brightness in a frame of a motion picture. If theluminance is high, the picture is bright and if it is low the picture isdark. Thus, luminance is the black and white part of the picture.

This luminance profile is used to find out that series of nine pixels 34out of the previous frame 35 which fits best with the series of ninepixels 30 of the current frame 32. In the embodiment of FIG. 5 theluminance profile of the series of nine pixels 30 of the current frame32 are compared with the luminance profiles of several correspondingseries of nine pixels 34 of the previous frame 35. In order to derivethe true motion the series of nine pixels 30 will be shifted over thesearch range in the horizontal direction 36. It is assumed that thatseries of nine pixels 34 of the previous frame 35 which shows the bestluminance profile matching (with the series of nine pixels 30 of thecurrent frame 32) is the correct series of pixels. These series ofpixels 30, 34 are then used for the computation of the motion vector.

A typical value for the search range is, e.g. 64 pixels (+31 . . . −32).However, it may also be possible to use less than 64 pixels, however,then the quality of the result of this comparison is increasingly goingdown. On the other hand it is also possible to use more than 64 pixels.Then the quality of the selection result is going up, however, thisneeds more computational effort. Therefore, typically a trade-off whichprovides an optimization between best quality of the selection resultand simultaneously a minimum computation effort is employed.

In a preferred embodiment for each selected motion vector 37 a singlematching process is performed in the way described above. This matchingprocess is performed by assigning a quality degree and/or a failuredegree for each series of pixels 30. Then, a quality-degree and/or afailure degree are assigned to each one of those series of pixels 30which are undergoing the matching process. Those series of pixels 30having the highest quality-degrees and/or the lowest failure degrees areselected as most probable series of pixels. These series of pixels 30are then used for computing the motion vectors for the horizontalmotion. Typically, but not necessarily a SAD method (SAD=sum of absolutedifference) and/or ADRC (Adaptive Dynamic Range Coding) method is usedfor the comparison of the luminance and/or chrominance values.

Using Pre-Selected Motion Vector Samples for the Motion Estimation:

Assuming the motion of an object in the scene will be constant fromfield/frame to field/frame and the object is larger than a series ofpixels (e.g. the above mentioned 9 pixels), then the matching processcan then be performed more efficiently if a set 38 of pre-selectedmotion vectors 37—the so-called motion vector samples 37—are checked fora matching of the luminance profile (see FIG. 5). For example, oneselected motion vector 37 can be taken from the neighbouring pixel. Asecond selected motion vector can be taken from the previous line, ifthe already estimated motion vectors are stored in a vector memoryspecially designed for the different motion vector samples.

The zero-vector which indicates no motion of the object is typically oneof the most used motion vector samples. This zero-vector is used inorder to more efficiently detect regions within a picture showing nomotion. In principle the amount of pre-selected motion vectors 37 whichwill taken into account depend strongly on what kind of motion vectorquality is desired.

Variation of Selected Motion Vectors:

In order to set up the process of motion estimation and to follow thedeviation from the constant motion, a variation of certain pre-selectedmotion vectors is required for test operation purposes. That means thatfor pre-selected motion vector samples a certain amount of motion willbe added or subtracted. This can be done by applying a variance withdifferent amount of motion speed to these motion vectors. The testedimplementation checks between odd pixels and even pixels alternating anupdate of +/−1 pixels and +/−4 pixels on the previously determinedmotion vector. The selection of the variance is adjustable and variableas required or as the need arises and depends e.g. on the resolution ofthe incoming video signal.

For the line-based motion estimation it is very advantageous that themotion vector will converge quickly for the real motion in the scene.Therefore, the selection of the tested motion vectors is treateddifferently for the first line of a frame or field. For the first lineof a frame or field testing is not possible in the normal way since aline above the first line which is needed for testing is not existing.In the first line of each field the selected motion vectors whichnormally test the motion vectors of the line above are loaded withvector values, which e.g. vary according to a triangle function frompixel to pixel. The triangle function oscillates between an adjustableminimum value and an adjustable maximum value. For that purpose alsoother regular oscillating functions e.g. a saw tooth function, asinusoidal function, and the like may be employed for the determinationof the motion vector of the first line.

The Matching Process:

In a preferred embodiment the matching process assigns a failure valueto each tested motion vector. In another embodiment this value may bealso a quality value. It might also be possible to evaluate as well afailure value and a quality value for the matching process. Preferably,the sum of the absolute difference (SAD) is used as the failure value orto at least derive the failure value. Ideally, to find the optimalmotion vector a failure value of zero is needed. However, typically thefailure value is different from zero. Therefore, the motion vectorcorresponding with the lowest failure value is then selected as the mostprobably motion vector representing the motion of an object in the localscene.

Attenuation of the Vector Selection, Vector Damping:

In a preferred embodiment a damping value is used which depends on thevector attenuation of the different motion vectors. This enables tocontrol the motion vectors with equal failure values and/or to furnishthe motion vector selection process with a certain direction.

Vector Memory:

The different motion vectors are advantageously stored in a vectormemory. These motion vectors can be then—if required—fetched from thevector memory for further processing and/or for the motion estimation ofthe next pixels.

The motion estimation process forms a recursive process. Therefore, thesize of this vector memory mainly depends on the desired quality levelof the matching process. In one embodiment, the tested implementationcomprises only one line of a vector memory. In this vector memory everysecond motion vector will be stored alternately, in order that an accessof the motion vectors from the measured line above is possible.

Robustness Improvement by Providing a Vector Histogram:

In a preferred embodiment a motion vector histogram is calculated inorder to create a highly reliable and homogeneous field of motionvectors. This vector histogram allows a vector majority ranking toderive most and less used motion vectors in an actual scene.

FIG. 6 shows an example to illustrate the basic principle for theprovision of a motion vector histogram. FIG. 6 shows a vector histogramgenerator 40 to provide a motion vector histogram. In the embodiment inFIG. 6 the vector histogram generator 40 comprises a switching device41, which is controlled by an +1-incrementing device 42. The switchingdevice 41 is controlled on the one hand by a motion vector 43information and on the other hand by the incrementing device 42 whichshift the switching device 41 to the next input terminal of a countingdevice 45 whensoever the next identical motion vector 43 occurs. Thecounting device 45 which comprises different counter cells 44 counts theoccurrence of each motion vector and increments the counter by +1 foreach occurrence of the motion vector. A ranking device 46—which e.g.comprises a simple comparing means—is coupled to the output terminals ofthe different counter cells 44 of the counting device 45. This rankingdevice 46 selects the most often used motion vector and applies thismotion vector for the estimation determination. The most often usedmotion vector may be then stored in a motion vector histogram memory 47.

The provision of a motion vector histogram can be done either for thewhole frame or field or only for parts of the frame or field. It is veryefficient to split the picture into horizontal stripes and return a mostoften used vector for each stripe. In very a preferred embodiment newsticker information within a picture can be detected in that way veryreliable.

Unreliable Vector Detection:

Under certain circumstances the motion estimation will not deliverreliable motion vectors. Especially for vertical movements as well asfor occlusions and uncovered areas the matching process sometimes doesnot provide a reliable access of the current and previous line. Notreliable motion vectors, however, lead to relatively large, undesireddifferences for the compensation process.

FIG. 7 shows an example of a flow diagram to illustrate the process forthe detection of unreliable vectors. In this detection process theluminance pixel value 50 of the vector compensated previous frame orfield is compared 52 with the luminance pixel value 51 of the vectorcompensated current frame or field. If this difference 52 stronglyexceeds an adjustable threshold value 53 the current motion vector willbe selected as a bad vector 54. Otherwise, it is assumed that thisvector is a reliable vector 55. This bad vector 54 will be marked 56 andthe MSB-bit of the vector memory will be set e. g. to one. The MSB-bitthen indicates whether this vector can be used for further estimationprocessing or not.

Bad Vector Replacement:

FIG. 8 shows an example of a flow diagram to illustrate the searchprocess for a bad vector replacement and an error concealment. In thissearch process bad motion vectors are treated in a special way in orderto avoid introducing artefacts. In case of the occurrence of a badvector, a reliable vector will be fetched from the memory e. g. for amotion vector for the line above. Therefore, preferably only thosemotion vectors are used in this search process which are not marked asunreliable (see above with regard to FIG. 7).

The search process starts e.g. from the most outer border of the searchrange in order to find a reliable vector of the surrounding area. If thebad vector signal 60 indicates no bad vector 61 then this motion vectoris used for the further motion estimation process 62. Otherwise, asearch process 63 is started to select a motion vector from previousestimation processes. If this search process 63 indicates a bad vector64 then this bad vector is replaced by a reliable vector 65 (see alsosteps 53-55 in FIG. 7). After replacing the bad vector against areliable vector this reliable vector is then used for the motioncompensation process. Otherwise, it is checked if the end of the searchrange 66 is reached. If the end of the search range 66 is reached theoutput signal is blurred 67. If the end of the search range 66 is notyet reached the search process returns 68 to the step 63.

Adaptive Error Region Extension and Error Spike Suppression:

The remaining unreliable vectors typically lead to artefacts in theinterpolated picture. In one embodiment—as already outlined above withregard to FIG. 8—these artefacts can be covered by an intentionalblurring of this artefact comprising regions. In a preferred embodimentthese artefact comprising regions—hereinafter also referred to as errorregions—are cleaned up from single spikes in order to provide efficientartefact concealment. This process is shown with regard to FIG. 9. FIG.9 shows by means of three illustrations (A)-(C) examples to illustratethe artefact concealment according to the present invention.

The artefact concealment according to the present invention may beperformed on the basis of an error spike suppression and/or an errorregion extension method. In FIGS. 9(A)-(C) the left diagram side denotesthe vector samples before the artefact concealment and the right diagramside denotes the vector samples after the artefact concealment. Thereliable motion vectors are denoted by reference number 70 and theunreliable motion vectors which additionally comprise an arrow toillustrate the spike are denoted by reference numbers 71, 72, 73.

Single unreliable vectors 71—hereinafter also referred to as spikes—willbe suppressed (FIG. 9(A)). Double spikes 72 will be not suppressed (FIG.9(A)). Double spikes 73 intersected by a single reliable motion vector70′ will be combined to form one uniform region of three spikes 74 (FIG.9(B)). In a preferred embodiment, after cleaning up the spikes 71-73 inthat way, the remaining error regions 75 are extended additionally by anamount of bad vectors 76 on both sides of this error region 75 (FIG.9(B)). Typically two spikes 76 are added on both sides of the errorregion 75.

This artefact concealment operation ensures a proper behaviour of theblurring filter. In a preferred embodiment, the blurring filter is usedonly for blurring operations in the horizontal direction.

An error spike denotes the single occurence of one error surrounded by amostly error-free region.

It is self understood that the above mentioned numerical data is merelyillustrative and may be adapted to best provide an optimized blurringeffect.

FIG. 10 shows a block diagram illustrating an embodiment of theline-based motion estimation according to the present invention asdescribed above and as implemented in a motion estimation device 25 asshown in FIG. 4.

The motion estimation device 25 comprises a matching device 80, acost/quality function device 81 and a vector selector device 82, whichare arranged in series connection between the input side 83 of themotion estimation device 25 where the image data signals X1, X1′ storedin the both line memories 23, 24 are provided and the output side 84 ofthe motion estimation device 25 where the motion vector signal X4 inpresent. In the device elements 80-82 a matching process and a vectorselection as described with regard to FIG. 5 is implemented.

The motion estimation device 25 further comprises a vector qualitydevice 85 which is connected on the one hand to the input side 83 and onthe other hand to the output side 84. The vector quality device 85generates a quality signal X6 comprising an information of the vectorquality out of the image data signals X1, X1′ and the motion vectorsignal X4.

The motion estimation device 25 further comprises a vector histogramdevice 86 and a vector majority device 87 which are arranged in seriesconnection in a feedback path between the output side 84 and thematching device 80. Here, in the device elements 86, 87 a vectorhistogram is generated to provide a ranking of most and less usedvectors in the actual scene as shown and described with regard to FIG.6. Thus, the elements 86, 87 correspond to the vector histogramgenerator 40 of FIG. 6.

The motion estimation device 25 may further comprise a further linememory 88 to store the motion vector data X4 and/or the data X6 for thevector quality.

The motion estimation device 25 further comprises a vector sample device89.

This vector sample device 89 is also arranged in the feedback path andis connected at its input side with the line memory 88, the vectormajority device 87 and advantageously with a further device 90. Thisfurther device 90 performs a variation of the motion vector samples byusing a special signal having a certain magnitude, e.g. a sinusoidalsignal, a saw tooth signal or the like. This certain signal is then usedfor a testing and/or matching process and/or an up-dating process of thefirst line of a frame or field. However, it might also be possible torandomly up-date different lines of the frame or field. On its outputside, the vector sample device 89 is connected at its output side to thematching device 80.

The motion estimation device 25 further comprises a vertical motionestimation device 91. For vertical motions the above describedone-dimensional motion estimation algorithm is not able to compensatefully motion in the vertical direction. However, the occurrence ofvertical motions can be used to reduce the compensation in same regionsof the picture by splitting the picture into different regions to derivevertical motion for each region. In this case the luminance values ofthe lines in the different region of the same picture will be summed upand stored individually for each line of this picture. This results inan accumulated vertical profile for different regions of the samepicture. Then, the whole picture can be divided into smaller regions toderive a vertical motion for each of these regions. This vertical motionestimation process is performed in the vertical motion estimation device91 which is connected to the input side 83 and which provides at itsoutput side a sector based vertical motion index X7.

Thus, the vertical MEMC as sketched above can be performed independentlyof horizontal MEMC and also in combination with the horizontal MEMC,wherein the combination can be performed in dependence on a certainsituation or the motions present, respectively. Further, such amethodology allows an implementation of vertical MEMC, which does notneed large amounts of additional memory capacity to analyze data ofconsecutive frames being the case in the most methodologies of the priorart.

The motion estimation device 25 further comprises a vector dampingdevice 92. In this damping device 92 a damping value as described abovemay be used to damp vector samples of the vector sample device 89 and toprovide damped vector samples to the vector selector 82.

Hereinafter the motion compensation process which is performed in themotion compensation device 26 of FIG. 4 is described with regard to FIG.11 in more detail. FIG. 11 shows a block diagram illustrating anembodiment of the line-based motion compensation according to thepresent invention using adaptive artefact concealments as describedabove.

The motion compensation device 26 comprises a compensation device 100which performs the temporal motion interpolation according to the motionvectors X4 estimated by the motion estimation device 25. In a preferredembodiment the compensation device 100 comprises a Median Filter whichuses as input data the luminance values of the vector compensatedprevious line, the vector compensated current and the uncompensatedprevious line. Additionally, also the chrominance values can becompensated.

Depending on the vector quality a replacement vector indicated asreliable vector will be searched in the local area of the vector memoryfrom the line above. If no reliable vector can be found the adaptiveblurring or an other fallback methodology typically tries to cover thisartefact.

The motion compensation device 26 further comprises a vertical motioncontrol device 101 which provides a control signal X8 to thecompensation device 100 in order to incorporate also information of avertical motion to the compensation device 100.

The motion compensation device 26 further comprises a bad vectormodification device 102. Based on information X4, X6 provided by themotion estimation device 25 the bad vector modification device 102modifies bad vectors as shown and described above with regard to FIGS. 8and 9. These information X9 about modified bad vectors is thenused—together with the control signal X8—to perform the motioncompensation within the compensation device 100. The compensation device100 then generates at its output side a motion compensated image datasignal X10.

The motion compensation device 26 further comprises an adaptive blurringdevice 103. Based on the motion compensated image data signal X10 and ablurring control signal generated by the bad vector modification device102 this adaptive blurring device 103 performs an adaptive blurring e.g.such as described with regard to the FIGS. 8 and 9. The adaptiveblurring device 103 generates an adaptive blurred image data signal X5′which might correspond to the image signal X5 of FIG. 4.

FIG. 12 shows a block diagram of a second embodiment of a line-basedMEMC system according to the present invention using the line memoriesassigned to the de-interlacer device also for the motion estimationdevice.

Unlike the first embodiment in FIG. 4 a de-interlacer device 113 isarranged between the line memories 110, 111, 112 and the motioncompensation device 26. The de-interlacer device 113 is typically usedto convert a field represented by video data stream into a full framewhich is then also represented by another video data stream.

On-chip solutions for video processing which are memory-based havealready existing internal line buffers 110-112—the so-called linememories 110-112—which carry video data from the previous and currentfield or frame. These line buffers 110-112 can be located e. g. withintemporal noise reductions or de-interlacing units 113 which operatemotion adaptive. With the proposed line-based MEMC these line bufferscan be reused additionally for the motion estimation. For that purposeand in order to reduce motion judder artefacts from movie sources, amovie detector which indicates the current interpolated sequence ofpull-down mode is used. A line buffer selector transfers the videosignal data to the motion estimation device according to the previousand the current video input signal. This technique allows using alreadyexisting memory resources also for motion estimation which also preventsadditional bandwidth for the temporal up-conversion process. Therefore,the chip area for the motion estimation and the motion compensation canbe reduced to a minimum.

The de-interlacer device 113 uses three line memories 110, 111, 112coupled on their input side to the memory bus 22 and providing at theiroutput side line data. This line data provided by the line memories 110,111, 112 is processed within the de-interlacer device and then providedto the motion compensation device 26. According to the presentinvention, these line memories 110, 111, 112 are additionally used alsofor the motion estimation device 25. For this purpose, the system 20additionally comprises a selector device 114, where a movie sequence X0is provided to this selector device 114. This movie sequence X0 may bethen stored in an external memory 28 via the memory bus 22 and can beread out from this external memory 28 through the line memories 110,111, 112. For an IMC operation, this data stored in the line memories110, 111, 112 of the de-interlacer device 113 can be also used for MEMC.For this purpose the data stored in the line memories 110, 111, 112 isthen provided as well to the motion estimation device 25 and the motioncompensation 26 device.

FIG. 13 shows a block diagram of a third embodiment of a line-based MEMCsystem according to the present invention.

Unlike the second embodiment in FIG. 2 this third embodiment combinesthe line-based MEMC with non-vector based-up conversion methods likeMCSFI and GFI. FIG. 24 shows a block diagram for the combination of linebased MEMC with non-vector based up-conversion methods like MCSFI andGFI.

GFI (Grey Frame Insertion) denotes an up-conversion method to double theincoming frame rate of a video sequence. MCSFI (Motion Controlled SmoothFrame

Insertion) denotes an up-conversion method which is motion detectorcontrolled. Instead of these up-conversion method also a SFI-method(Smooth Frame Insertion) and/or a DFI-method (Dynamic Frame Insertion)may be applied, whereas SFI stands for an up-conversion method to doublethe incoming frame rate of a video sequence and DFI stands for anup-conversion method to double the incoming frame rate of a videosequence.

Typically for those up-conversion methods the high contrast regions aredetected per unit and excluded from the compensation processing due tothe incapability of those methods to compensate within these regions.Therefore a pixel gain detector controls a fader, which normally fadesover from the processed video signal to the unprocessed video signal inorder to reduce the introduction of artefacts. The present invention ofthe line-based MEMC is capable to perform the up-conversion also inthese high contrast regions. Therefore, the signal of the line-basedMEMC is applied to the unprocessed fader input of the non-vector-basedup-conversion methods.

While embodiments and applications of this invention have been shown anddescribed above, it should be apparent to those skilled in the art, thatmany more modifications (than mentioned above) are possible withoutdeparting from the inventive concept described herein. The invention,therefore, is not restricted except in the spirit of the appendingclaims. It is therefore intended that the foregoing detailed descriptionis to be regarded as illustrative rather than limiting and that it isunderstood that it is the following claims including all equivalentsdescribed in these claims that are intended to define the spirit and thescope of this invention. Nor is anything in the foregoing descriptionintended to disavow the scope of the invention as claimed or anyequivalents thereof.

It is also noted that the above mentioned embodiments, examples andnumerical data should be understood to be only exemplary. That meansthat additional system arrangements and functional units and operationmethods and standards may be implemented within the MEMC-system.

1. A method for line-based motion estimation and motion compensation invideo image data, comprising the steps of: providing a video signalcomprising video image data of a video line or part of the video line ofthe picture; buffering the video image data in at least one line memoryhaving the size of one video line or at least of the incoming oractually processing video image data.
 2. The method according to claim1, wherein the method comprises the steps of: performing the motionestimation by detecting and analysing the video image data and byderiving a motion vector depending on the detected motion, wherein themotion vector contains only motion data for motion of an object in onedirection in the horizontal direction; using the motion vector formotion compensation to interpolate a picture.
 3. The method according toclaim 2, wherein image data of the previous frame is derived from afirst line memory and image data of the current frame is derived from asecond line memory.
 4. The method according to claim 3, wherein thefirst line memory and/or the second line memory is/are further used in ade-interlacer application and/or a temporal noise reduction application.5. The method according to claim 4, wherein for the motion estimation amatching process is employed, wherein the luminance profile and/orchrominance profile is/are used as matching parameter.
 6. The methodaccording to claim 5, wherein the matching process comprises the stepsof: providing a first series of pixels within a line of a current frameor field; providing several second series of pixels within a line of aprevious frame or field around a given centre pixel for which the motionshall be determined; successively comparing luminance and/or chrominancevalues of the second series with the first series; assigning a qualitydegree and/or a failure degree to each second series; selecting thatsecond series having the highest quality-degrees and/or the lowestfailure degrees to be the most probable second series; computing themotion vector in the horizontal direction on the basis of the selectedsecond series.
 7. The method according to the claim 6, wherein aSAD-based method and/or an ADRC-based method are employed for thecomparison of the luminance and/or chrominance values.
 8. The methodaccording to claim 7, wherein pre-selected samples of motion vectors areused for the motion estimation.
 9. The method according to claim 8,wherein a variation of at least one selected motion vector is performedin order to set up the process of motion estimation and/or to follow thedeviation from the constant motion.
 10. The method according to claim 9,wherein the selection of the selected motion vectors is treateddifferently for the first line of a frame or a field compared to theother lines of the same frame or field.
 11. The method according toclaim 10, wherein a damping value which depends on the selected motionvector is used to adapt motion vectors with similar failure valuesand/or to bring the selection process into a desired direction.
 12. Themethod according to claim 11, wherein a motion vector histogram is usedto provide a ranking of the motion vector on the basis of most and lessused motion vectors in a current frame or field.
 13. The methodaccording to claim 12, wherein for detecting unreliable motion vectorsthe method comprises the steps of: Comparing a luminance pixel value ofa motion vector compensated previous frame or field with the luminancepixel value of a motion vector compensated current frame or field;Selecting the current motion vector as an unreliable vector if theevaluated difference in the comparing step exceeds a given thresholdvalue; Marking the selected unreliable vector and/or replacing theunreliable vector by a reliable vector.
 14. The method according to theclaim 12, wherein artefacts generated by not replaced unreliable vectorsare covered by an intentional blurring of those region which comprisesthis artefact.
 15. The method according to claim 14, wherein anadditional vertical motion estimation and motion compensation process isperformed independently of the horizontal motion estimation and motioncompensation process and/or in combination with the horizontal motionestimation and motion compensation process.
 16. An apparatus forline-based motion estimated and compensated field rate up-conversion inconsecutive frames or fields of a motion picture, wherein the apparatuscomprises: a processing unit configured to provide a video signalcomprising video image data of a video line or part of the video line ofa picture; and a memory unit coupled to the processing unit to bufferthe video image data.
 17. The apparatus according to claim 16, whereinthe apparatus comprises a first line memory for storing image data ofthe previous frame and a second line memory for storing image data ofthe current frame.
 18. The apparatus according to claim 17, wherein thefirst line memory and/or the second line memory are configured to befurther used in a de-interlacer device and/or a temporal noise reductiondevice.
 19. The apparatus according to at claim 18, wherein theapparatus further comprises: an input terminal for providing a videosignal comprising video image data; a buffer for buffering the videoimage data, wherein the buffer comprises at least one line memory havingat least the size of one video line or at least of the incoming oractually processing video image data; a motion estimation andcompensation device for performing the line-based motion estimation andmotion compensation; an output terminal for providing a motioncompensated video output signal.
 20. The apparatus according to claim19, comprising a third line memory, to store at least one motion vector.21. The apparatus according to claim 20, wherein the motion estimationand compensation device further comprises: A motion estimation devicewhich generates a motion vector X4 out of the image data stored in thefirst and second line memories; A motion compensation device whichperforms a motion compensation using the image data stored in the firstand second line memories and employing the vector data.
 22. A computerprogram product comprising a computer usable medium having a computerreadable program code embodied therein, said computer readable programcode being configured to be executed to implement a method forline-based motion estimation and motion compensation in video imagedata, comprising the steps of: providing a video signal comprising videoimage data of a video line or part of the video line of the picture;buffering the video image data in at least one line memory having thesize of one video line or at least of the incoming or actuallyprocessing video image data.
 23. The computer program product of claim22 wherein the method further comprises: e performing the motionestimation by detecting and analysing the video image data and byderiving a motion vector depending on the detected motion, wherein themotion vector contains only motion data for motion of an object in ahorizontal direction: using the motion vector for motion compensation tointerpolate a picture.